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Abstract-  This  paper  details  a  method  involving  a  series  of  metrics 
and  visuals  designed  to  assist  in  the  determination  of  a  single 
design  or  set  of  designs’  valuable  changeability  in  the  multi-epoch 
domain  of  Epoch-Era  Analysis  (EEA).  A  brief  introduction  to 
the  necessary  concepts  of  EEA  is  included,  with  references  for 
further  information.  Example  application  of  these  new  metrics  is 
provided  in  the  form  of  a  partial  case  study  of  a  potential  orbit¬ 
realigning  space  tug  system.  The  metrics  are  demonstrated  to 
give  insight  into  the  value  added  by  changeability  and  the  total 
value  provided  by  changeability  and  robustness  in  tandem. 

Reference  [12]  is  a  companion  paper,  detailing  additional  metrics 
and  methods  used  to  value  system  changeability  at  the  era  level. 
Multi-epoch  analysis  is  best  suited  for  understanding  the  performance 
of  systems  across  the  space  of  potential  future  uncertainties  when 
considering  their  ability  to  change  design;  era  analysis  uncovers 
additional  time-dependent  information  related  to  lifetime  value  and 
applied  change  mechanism  usage. 

Keywords-  changeability;  flexibility ;  metrics;  robustness; 
strategy;  tradespace  exploration;  valuation 

I.  Introduction/Motivation 

Changeability  (like  many  of  the  so-called  “ilities”)  is  a 
system  property  that  provides  a  means  to  the  end  of  lifetime 
value  delivery.  In  particular,  changeability  corresponds  with 
the  ability  to  alter  either  the  physical  design  parameters  or 
operations  of  the  system  and  can  be  leveraged  in  any  of  the 
lifecycle  phases  of  common  engineering  systems:  design,  build, 
integration  and  test,  and  operate.  For  example,  the  ability  to 
quickly  redesign  a  particular  subsystem  of  a  rocket  in  the  event 
of  a  requirements  update  would  represent  design-phase 
changeability,  whereas  the  ability  to  bum  fuel  and  adjust  orbit 
altitude  would  correspond  to  operations-phase  changeability. 

Changeability  is  experiencing  an  increase  in  interest  as 
engineering  systems  grow,  both  in  budget  size  and  system 
lifetime,  demanding  more  emphasis  on  value  delivery  over 
time  and  under  different  contexts.  The  apparent  potential 
performance  advantages  of  changeable  systems,  the  difficulty 
of  designing  a  system  fully  robust  to  changes  experienced  over 
decades,  and  the  increased  cost  of  failure  are  driving  the 
popularity  of  changeability  and  related  concepts.  For  example, 
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one  concept  with  many  similarities  to  changeability  is 
flexibility ;  indeed,  the  words  are  often  used  interchangeably. 
Saleh  et  al.  [1]  performed  a  survey  of  the  use  of  “flexibility”  in 
the  literature  for  different  fields,  mainly  managerial, 
manufacturing,  and  engineering  design,  while  cataloguing  the 
different  meanings  of  the  usage.  Focusing  here  on  the  meaning 
for  engineering  systems,  Saleh  finds  two  distinct  uses:  one  for 
flexibility  in  the  design  process  and  another  for  flexibility  in 
the  design.  Even  those  subtypes  of  flexibility  have  seen 
different  usage  of  the  word,  such  as  design  process  flexibility 
being  applied  to  both  customers  (flexibility  in  requirements 
specified)  and  designers  (flexibility  in  constraints  imposed). 
In-design  flexibility  is  similarly  split  amongst  various 
definitions,  although  most  relate  quite  directly  to  the  ability  of 
the  system  to  perform  different  functions,  and  it  is  this  type  of 
flexibility/changeability  that  this  paper  addresses. 

This  research  seeks  to  utilize  the  Epoch-Era  Analysis 
(EEA)  framework  as  a  platform  on  which  to  properly  value  the 
changeability  of  engineering  systems.  The  end  goal  is  that  this 
will  allow  system  designers  to  justify  investment  in 
changeability-enabling  features  and  consider  changeable 
designs  on  an  equal  footing  with  more  traditional  and 
conservative  passively  value  robust  designs. 

II.  Epoch-Era  Analysis 

Epoch-Era  Analysis  is  a  system  design  approach,  developed 
by  Ross  and  Rhodes  [2],  designed  to  clarify  the  effects  of  time 
and  context  on  the  value  of  a  system  in  a  structured  way.  The 
base  unit  of  time  in  the  method  is  the  epoch ,  which  is  a  period 
of  time  defined  by  a  fixed  set  of  variables  describing  the 
context  in  which  the  system  operates.  These  variables  can 
encompass  any  exogenous  circumstances  that  have  an  effect  on 
the  usage  and  value  of  the  system:  weather  patterns,  political 
scenarios,  financial  situations,  operational  plans,  and  the 
availability  of  other  technologies  are  all  potential  epoch 
variables.  The  complete  set  of  epochs,  differentiated  using 
these  variables,  can  then  be  assembled  into  eras ,  ordered  sets  of 
epochs  creating  a  description  of  a  potential  progression  of 
contexts  over  time.  This  framework  provides  an  intuitive  base 
upon  which  to  perform  analysis  of  value  delivery  over  time  for 
systems  under  the  effects  of  changing  circumstances  and 
operating  conditions,  an  important  step  to  take  when  evaluating 
large-scale  engineering  systems  with  long  lifespans. 
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Epoch-Era  Analysis  was  created  with  the  intent  for  it  to  be 
used  in  conjunction  with  Multi-Attribute  Tradespace 
Exploration  (MATE),  which  models  large  numbers  of  designs 
and  compares  their  utilities,  typically  represented  as 
combinations  of  nonlinear  functions  of  performance  attributes 
[3].  MATE  is  a  powerful  tool  for  conceptual  system  design, 
allowing  for  the  evaluation  and  comparison  of  many  different 
potential  designs  that  could  be  chosen  for  building  and  fielding. 
A  design  vector  enumeration  is  used  to  define  many  potential 
systems  that  are  then  modeled  via  a  computer  simulation, 
allowing  for  a  more  complete  exploration  of  the  entire  design 
space  than  the  traditional  engineering  practice  of  fleshing  out  a 
handful  of  potential  designs  and  selecting  from  among  them. 

In  addition  to  its  function  as  a  temporal  extension  of  the 
typically  static-context  field  of  tradespace  exploration,  Epoch- 
Era  Analysis  can  be  used  as  a  framework  for  considering 
value-over-time  regardless  of  the  underlying  methodology. 
Treating  the  passage  of  time  as  a  stochastic  sequence  of  static 
conditions  can  be  used  to  extend  other  common  engineering 
practices,  including  the  investigation  of  a  single  point  design 
for  which  time-dependent  performance  variables  are  present. 
This  allows  for  a  broader  application  of  Epoch-Era  Analysis  to 
different  studies. 

III.  Complications  in  Valuing  Changeability 

Developing  metrics  for  properly  valuing  changeability  is  a 
complicated  task.  There  are  a  few  criteria  that  are  beneficial 
for  metrics  to  possess  in  order  to  improve  their  ability  to 
accurately  rate  and  compare  alternatives,  but  which  many 
previously  created  changeability  metrics  do  not  satisfy.  One  of 
these  is  the  universality  criterion,  which  stipulates  that  a  score 
of  X  for  the  metric  is  equally  as  good  or  bad  as  a  score  of  X  in 
a  different  context.  Without  this  distinction,  it  is  extremely 
difficult  to  effectively  use  the  metric  to  address  problems  of 
uncertainty.  In  the  case  of  EEA,  scores  in  a  non-universal 
metric  become  incomparable  across  different  epochs.  This  is  a 
key  problem  for  any  metrics  that  use  multi-attribute  utility  as  a 
value  statement,  as  MAU  is  neither  on  a  ratio  scale  nor  ordinal 
between  epochs  under  different  preferences,  making  a  score  of 
0.3  not  equivalent  to  the  same  score  in  a  different  epoch. 
Previous  valuable  changeability  metrics  such  as  Value- 
Weighted  Filtered  Outdegree  [4]  suffer  from  this  weakness. 

Another  useful  quality  for  metrics  to  possess  is  for  their 
scores  to  be  independent  of  the  set  of  designs  being  considered. 
If  a  design’s  score  is  a  function  of  the  other  designs  in  the 
tradespace  or  in  a  point-design  study,  the  use  of  that  metric 
adds  a  very  large  burden  of  proof  on  the  design  team  to  show 
that  any  “good”  design  is  good  regardless  of  the  other  options 
under  consideration.  This  is  particularly  important  for  metrics 
valuing  changeability,  as  many  stakeholders  require  concrete 
evidence  of  the  added-value  gains  of  changeability  options  in 
order  to  be  convinced  to  fund  them.  Independence  is  also  a 
useful  property  even  for  applications  where  value  relative  to 
alternatives  is  acceptable,  as  it  allows  the  set  of  designs  of 
interest  to  be  modified  during  analysis  while  maintaining 
metric  stability.  As  an  example  of  a  project  where  relative 
value  is  acceptable,  think  of  a  launch  vehicle  selection  for  a 
frilly  completed  payload  cleared  for  deployment:  there  is  a 
fixed  set  of  existing  alternatives  for  launch  vehicle  and  one 


must  be  selected  (the  project  is  not  optional  as  the  payload  must 
be  put  into  orbit).  There  is  no  need  to  prove  a  baseline 
“goodness”  for  the  selected  vehicle  as  long  as  it  is  superior  to 
the  others,  because  all  alternatives  are  included  in  the  decision 
and  the  decision  to  choose  none  is  not  an  option.  Under 
conditions  like  these,  independent  metrics  are  less  critical  but 
still  superior  to  dependent  metrics. 

Changeability  also  has  a  fundamentally  two-dimensional 
value.  The  value  of  changeability  is  derived  from  two  distinct 
sources:  (1)  the  increase  in  system  value  resulting  from  a 
change,  as  well  as  (2)  the  number  of  options  available, 
generating  robustness  to  perturbations  via  breadth  of  choice 
and  redundancy.  This  dichotomy  between  magnitude  and 
counting  value  is  frequently  in  tension,  resulting  in  metrics  that 
account  for  only  one  of  the  two  sources.  For  example,  Filtered 
Outdegree  [5]  is  explicitly  designed  to  “count”  option  paths, 
and  its  extension  Value- Weighted  Filtered  Outdegree  does  not 
resolve  the  magnitude  value  as  previously  referenced. 
Similarly,  real  options  methods  [6,7]  frequently  simplify  the 
problem  into  a  “dollarized”  magnitude  value  for  particular 
scenarios,  which  both  ignores  the  counting  value  of  multiple 
transition  choices  and  is  inappropriate  for  many  large 
engineering  systems  that  do  not  deliver  value  in  the  form  of 
money.  Simultaneously  accounting  for  both  sources  of  value 
will  be  critical  in  creating  a  broadly  applicable  valuable 
changeability  metric. 

Understanding  all  of  these  complications,  it  is  important  to 
make  a  statement  about  the  nature  of  valuable  changeability: 
value  is  only  realized  from  changeability  through  executed 
changes.  This  is  a  simple  statement,  but  one  that  is  frequently 
ignored  in  favor  of  metrics  that  attempt  to  aggregate  the  value 
of  all  potential  changes.  A  crucial  step  then  before  valuing  a 
design’s  changeability  is  determining  how  the  design  will 
change  when  faced  with  any  particular  context.  To  this  end, 
we  define  a  strategy  that  specifies  how  the  system  stakeholder 
intends  to  utilize  any  available  changeability.  This  strategy  can 
range  from  the  simple  (maximize  utility  for  any  epoch  at  any 
cost)  to  the  complex  (execute  change  targeting  the  available 
design  with  highest  predicted  lifetime  value,  but  only  if  utility 
falls  below  a  certain  threshold  and  design  increases  in  cost 
efficiency,  and  in  certain  epochs  changes  are  not  allowed). 
Frequently,  it  will  be  of  interest  to  consider  multiple  strategies, 
as  the  different  strategies  can  be  compared  for  their  relative 
effectiveness  at  increasing  system  value.  Employing  a  strategy 
thins  out  the  multitude  of  possible  options  down  to  one  selected 
option  for  a  given  design  in  each  epoch,  and  it  is  this  selected 
transition  that  we  will  use  to  value  the  system’s  changeability. 
This  paper  will  refer  to  the  selected  option  of  design  d  as  <7*, 
where  <7*  is  defined  separately  for  each  epoch  and  can  be  equal 
to  d  if  the  design  does  not  change.  When  multiple  strategies 
are  under  consideration,  different  indicators  should  replace  *  to 
clarify  what  strategic  end  state  is  being  referenced. 

The  use  of  strategies  with  EEA  allows  the  reconciliation  of 
the  magnitude  and  counting  values  of  changeability.  The 
single  selected  path  from  each  design  in  each  epoch  will  be 
valued  for  its  magnitude:  the  benefit  gained  from  executing  that 
design  change.  The  counting  value  of  the  changeability  options 
for  a  design  comes  out  when  considering  the  entire  epoch 
space:  when  confronted  with  a  full  variety  of  changing 
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contexts,  designs  with  more  options  will  tend  to  perform  better 
in  more  of  those  contexts.  By  intelligently  probing  these  two 
sources  of  information,  a  design  team  can  extract  insight  about 
the  total  valuable  changeability  inherent  in  a  design. 

IV.  Multi-Epoch  Changeability  Value  Metrics 

A  set  of  metrics  designed  to  identify  the  value  contained  in 
these  strategically  selected  transitions  is  presented  in  the 
following  sections. 

A.  Fuzzy  Pareto  Number  (FPN) 

Pareto  efficiency  is  a  common  concept  in  tradespace 
exploration,  where  a  system  design  is  described  as  “Pareto 
efficient”  or  “on  the  Pareto  front”  if  it  is  not  dominated  in  both 
cost  and  utility  by  any  other  design.  We  can  also  consider 
designs  to  be  “fuzzily”  efficient,  by  allowing  them  to  be  within 
a  certain  distance  of  the  true  Pareto  front  [8].  This  distance  is 
calculated  as  a  percentage  of  the  range  of  the  cost/utility  data; 
for  example,  all  1%  fuzzy  Pareto  efficient  designs  will  be  less 
than  or  equal  to  1%  of  the  range  of  costs  more  and  1%  of  the 
range  of  utilities  less  than  a  truly  (0%)  Pareto  efficient  design. 
The  Fuzzy  Pareto  Number  (FPN)  of  design  d  is  the  smallest 
percentage  K  for  which  the  design  is  in  the  fuzzy  Pareto  set  PK. 

FPN(d)  =  min{  K\dczPK}  (1) 

FPN  is  a  measure  of  cost/utility  efficiency  calculated  for 
each  design  in  each  epoch  and  for  which  smaller  is  better,  with 
a  minimum  of  0  and  a  maximum  of  100.  It  will  be  used  as  an 
indicator  of  design  value  in  other  metrics.  Note  that,  unlike 
many  other  measures  of  value  such  as  multi-attribute  utility, 
FPN  is  universal  in  scale  (a  design  with  an  FPN  of  3  in  any 
epoch  is  within  3%  of  cost-efficiency).  FPN  is  also  largely 
independent  of  the  enumerated  design  space:  even  though  it  is 
defined  relative  to  the  Pareto  front,  which  in  a  tradespace  study 
is  determined  by  enumerated  designs,  multi-variable 
optimization  can  be  used  to  mathematically  explore  the  system 
dynamics  and  calculate  a  Pareto  front  for  a  point-design  study, 
regardless  of  the  point  designs  under  consideration  [9]. 
Similarly,  an  intelligent  enumeration  of  designs  in  a  tradespace 
will  adequately  approximate  the  exact  Pareto  front,  making 
FPN  insensitive  to  addition  or  removal  of  new  design  points. 

B.  Effective  (fuzzy)  Normalized  Pareto  Trace  (eNPT/efNPT) 

A  common  EEA  metric  for  identifying  passively  robust 
system  designs  is  the  Normalized  Pareto  Trace  (NPT),  which  is 
defined  as  the  percentage  of  epochs  in  the  epoch  space  for 
which  a  given  system  is  Pareto  efficient  [10].  This  can  be 
calculated  quite  easily  with  FPN;  it  is  simply  the  number  of 
epochs  in  which  a  design  has  an  FPN  of  zero,  divided  by  the 
total  number  of  epochs.  Similarly,  we  can  define  a  “fuzzy” 
corollary  (fNPT)  that  counts  all  epochs  with  an  FPN  less  than 
or  equal  to  a  certain  fuzziness  threshold.  Both  of  these  metrics 
describe  the  frequency  with  which  a  design  achieves  a  high 
level  of  value  across  all  potential  future  scenarios.  Note  that 
NPT  is  the  same  as  fNPT  with  a  fuzzy  factor  of  0%. 

With  the  definition  of  a  strategy,  we  now  know  that  some 
designs  will  choose  to  change  when  confronted  with  particular 
epochs.  Why  then  should  we  be  judging  designs  based  on  their 
own  FPN,  given  that  they  will  change  in  response  to  an  epoch 
shift?  Thus,  we  define  an  effective  version  of  the  above  two 


metrics  (eNPT,  efNPT)  which  considers  not  the  FPN  of  the 
design  d  itself  in  each  epoch,  but  the  FPN  of  that  design’s 
strategically  selected  end  state  d*  for  each  epoch.  This  allows 
designs  that  frequently  change  in  response  to  epoch  shifts  to  be 
graded  not  on  their  baseline  performance,  but  on  their 
changeability-enhanced  performance. 

eNPT{d)  =  [  Zepochs  1  {FPN(d*)=0}  ]  +  Nepochs  (2) 

efNPT(d,K)  =  [  Zepochs  1  {FPN(d*)<K}  ]  +  Nepochs  (3) 

A  design  that  scores  high  in  eNPT  or  efNPT  could  be  said 
to  be  “frequently  cost  efficient  across  the  space  of  potential 
future  scenarios  when  considering  its  planned  usage  of 
changeability ”.  Note  that  these  metrics  consider  passive 
robustness  and  changeability-enabled  robustness 
simultaneously,  as  passively  robust  designs  (which  will  not 
change  very  often  due  to  naturally  high  cost  efficiency)  will  be 
graded  on  their  own  FPN  for  most  epochs. 

C.  Fuzzy  Pareto  Shift  (FPS) 

While  the  previous  metrics  attempt  to  quantify  a  measure  of 
robustness  across  the  uncertainty  space  that  acknowledges 
changeability,  we  may  wish  to  clarify  the  magnitude  of  the 
value  of  a  design’s  selected  changes.  For  example,  two  designs 
may  score  identically  in  eNPT  but  derive  vastly  different  value 
from  their  respective  change  options  because  of  differing 
amounts  of  passive  robustness  versus  changeability.  To  do 
this,  we  analyze  the  Fuzzy  Pareto  Shift  (FPS)  distribution  of 
each  design. 

FPS(d)  =  FPN(d)  -  FPN(d*)  (4) 

FPS  is  defined  as  simply  the  difference  in  FPN  of  the  pre- 
and  post-change  states  (« d  and  d*)  for  a  given  design  in  a  given 
epoch.  Thus,  a  design  with  an  FPN  of  8  that  transitions  to  a 
design  with  an  FPN  of  2  would  have  an  FPS  of  8-2  =  +6.  The 
“shift”  in  Fuzzy  Pareto  Shift  represents  an  increase  or  decrease 
in  cost  efficiency  as  the  result  of  executed  changeability.  An 
increase  in  FPN  would  result  in  a  negative  FPS;  this  is  meant  to 
signal  a  loss  of  efficiency,  but  does  not  necessarily  signify  that 
a  mistake  has  been  made,  as  the  implementation  of  many 
strategies  (for  example,  utility  maximization)  will  sacrifice 
efficiency  for  gains  in  other  objectives.  An  epoch  in  which  no 
transition  is  made  will  have  an  FPS  of  zero,  as  the  initial  and 
final  states  are  the  same. 

Because  it  is  defined  separately  for  each  epoch,  a  design’s 
FPS  is  best  viewed  as  a  distribution.  This  distribution  curve 
can  be  compared  against  other  designs  for  an  intuitive 
understanding  of  the  relative  frequencies  of  different 
magnitudes  of  changeability  value  occurring  in  each  design 
across  the  epoch  space.  When  breaking  down  this  distribution 
into  representative  statistics,  preference  should  be  given  to 
order  statistics  (minimum,  maximum,  median,  percentiles)  over 
averages;  the  distributions  are  often  heavily  skewed  by  positive 
and/or  negative  outliers  which  makes  the  distribution  mean  ill- 
suited  to  summarizing  the  design  performance. 

D.  Available  Rank  Improvement  (ARI) 

Unlike  the  previous  metrics,  Available  Rank  Improvement 
(ARI)  does  not  depend  on  a  strategy,  but  rather  presupposes  an 
attempt  to  maximize  utility.  Here,  a  change  mechanism  is 
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defined  as  a  single  means  for  a  design  to  change;  for  example, 
a  modular  payload  bay  to  swap  payloads  and  thrusters  to  alter 
orbit  characteristics  are  two  potential  change  mechanisms  for  a 
satellite  system.  ARI  is  calculated  for  each  change  mechanism 
(r)  separately,  as  the  maximum  possible  improvement  in  utility 
rank-ordering  achievable  using  only  that  change  mechanism. 
The  term  dr  represents  all  available  designs  from  design  d  using 
only  mechanism  r. 

ARI{r,d)  =  Rank{d)  -  min {Rank(dr)}  (5) 

ARI  is  an  imperfect  metric,  as  it  requires  a  tradespace  and 
depends  heavily  on  the  chosen  enumeration  of  designs,  but 
serves  adequately  as  an  indicator  of  potential  achievable  value 
enabled  by  the  inclusion  of  a  particular  change  mechanism.  A 
“strategic”  version  of  ARI  can  be  calculated  by  swapping  out  dr 
with  the  strategic  end  state  d*  used  in  the  previous  metrics  and 
removing  the  min{}  function,  but  this  form  is  not 
recommended.  ARI  is  best  employed  to  represent  what  is 
made  “available”  by  a  change  mechanism,  and  the  other 
metrics  are  more  appropriate  for  evaluating  executed  strategic 
transitions. 

V.  Application  to  a  Space  Tug  Example 

A.  Introduction  and  Designs  of  Interest 

To  demonstrate  some  of  the  insights  that  can  be  gleaned 
from  these  metrics,  a  simple  case  study  is  now  presented.  The 
Space  Tug  data  set  is  a  conceptual  tradespace  study  for  a 
satellite  designed  to  realign  other  satellites  when  their  orbits 
begin  to  deviate  from  their  planned  trajectories.  It  is  a  small 
tradespace,  with  only  384  designs  being  considered  across  16 
epochs.  Large  studies  can  feature  over  100,000  designs  and 
1,000  epochs,  but  a  small  study  provides  the  opportunity  to 
quickly  assess  the  viability  of  changeability  metrics.  The 
designs  are  all  graded  on  three  attributes:  available  delta  V, 
speed,  and  mass  capability.  This  study  is  based  on  the  work  of 
McManus  and  Schuman  [11]. 

There  are  only  four  design  variables  used  to  generate  the 
Space  Tug  tradespace:  propulsion  type,  fuel  mass,  manipulator 
capability,  and  Design  for  Changeability  (DFC)  level.  In  this 
case,  DFC  level  is  implemented  as  a  mass  penalty  on  the 
satellite  and  serves  as  an  enabler  of  improved  or  additional 
change  mechanisms.  Think  of  DFC  as  the  inclusion  of  extra 
design  features  or  margin,  at  an  additional  cost,  for  example. 
DFC  is  implemented  in  three  levels  (0,1,2)  and  the  associated 
change  mechanisms  are  listed  in  Table  I.  As  described,  all 
designs  with  either  a  bipropellant  or  cryogenic  engine  can 
switch  between  the  two  options  and  all  designs  are  capable  of 
changing  fuel  tank  size,  and  both  of  these  options  reduce  in 
cost  with  any  investment  in  DFC.  Also  gained  from 
investment  in  levels  1  or  2  of  DFC  is  the  ability  to  switch 
manipulator  capability;  increasing  capability  improves  utility, 
but  decreasing  capability  reduces  mass  and  thus  cost.  Finally, 
the  DFC  level  2  designs  can  also  refuel  in  orbit,  which  extends 
lifetime  while  sparing  the  costs  of  redesigning  and  relaunching 
the  satellite. 

These  change  mechanisms  specify  which  other  designs  are 
accessible  via  a  change  for  each  design.  The  combination  of 
multiple  change  mechanisms  can  lead  to  even  further  designs, 
and  considering  these  multi-arc  transitions  leads  to  a  full 


accessibility  matrix,  which  indicates  all  available  end  states  via 
any  combination  of  change  mechanisms  for  each  design.  The 
Space  Tug  full  accessibility  matrix  is  shown  in  Figure  1,  with 
design  numbers  on  both  axes.  The  plot  is  read  by  locating  a 
design  number  on  the  vertical  axis  and  reading  across  to  find 
all  available  other  designs  on  the  horizontal  axis  as  indicated  by 
a  mark  in  the  appropriate  column.  This  plot  gives  a  fast 
understanding  of  how  connected  the  tradespace  is,  and  will  also 
qualitatively  allow  for  an  assessment  of  designs  with  many 
change  options  and  thus  the  potential  for  high  valuable 
changeability,  particularly  in  the  counting  value. 

Table  I.  Space  Tug  Change  Mechanisms 


No. 

Change  Mechanism 

Effect 

DFC  level 

1 

Engine  Swap 

Biprop/Cryo  swap 

0 

2 

Fuel  Tank  Swap 

Change  fuel  mass 

0 

3 

Engine  Swap  (reduced  cost) 

Biprop/Cryo  swap 

1  or  2 

4 

Fuel  Tank  Swap  (reduced  cost) 

Change  fuel  mass 

1  or  2 

5 

Change  Capability 

Change  Capability 

1  or  2 

6 

Refuel  in  Orbit 

Change  fuel  mass 
(no  redesign) 

2 

0  50  100  150  200  250  300  350 


nz  =  13568 

Figure  1 .  Space  Tug  Full  Accessibility  Matrix 


This  paper  does  not  address  the  process  of  selecting 
designs  of  interest,  but  it  is  usually  a  good  idea  to  narrow  the 
field  of  consideration  to  a  subset  of  the  entire  tradespace  before 
considering  valuable  changeability,  as  there  are  likely  to  be 
many  designs  that  are  uninteresting  to  the  stakeholders 
regardless  of  changeability.  Any  indicators  of  value  or 
screening  metrics  can  be  used  to  downselect  the  designs.  For 
this  study,  seven  designs  of  interest  were  selected  using  a 
combination  of  Normalized  Pareto  Trace,  frizzy  Normalized 
Pareto  Trace,  and  Filtered  Outdegree.  The  designs  and  their 
parameters  are  shown  in  Table  II. 

B.  Comparing  Designs  with  Multi-Epoch  Analysis 

Having  selected  designs  of  interest,  the  goal  is  now  to  use 
the  new  metrics  to  effectively  compare  their  valuable 
changeability.  In  the  Space  Tug  case  study,  there  are  16 
epochs  under  consideration,  representing  eight  different 
“preference  sets”  (utility  functions)  corresponding  to  different 
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TABLE  II.  Space  Tug  Designs  of  Interest 


Design  No. 

Ref 

Prop  Type 

DFC  Level 

Fuel  Mass  (kg) 

Capability  (kg) 

Speed 

Delta  V  (m/s) 

Base  Cost  ($M) 

1 

A 

Biprop 

0 

30 

300 

Fast 

143 

97 

29 

B 

Nuke 

0 

1200 

300 

Fast 

7381 

306 

47 

C 

Cryo 

0 

10000 

1000 

Fast 

6147 

628 

128 

D 

Nuke 

0 

30000 

5000 

Fast 

14949 

3020 

191 

E 

Nuke 

1 

10000 

1000 

Fast 

16150 

980 

328 

F 

Biprop 

2 

50000 

3000 

Fast 

4828 

2804 

376 

G 

Elec 

2 

30000 

5000 

Slow 

27829 

3952 

TABLE  III.  Space  Tug  eNPT/efNPT  Results  for  Designs  of  Interest 


Designs 

eNPT 

efNPT  (5%  fuzziness) 

Do 

Nothing 

(NPT) 

Max  U 

Max 

Eff 

Survive 

Do 

Nothing 

(fNPT) 

MaxU 

Max 

Eff 

Survive 

A 

0.75 

0 

0.875 

0 

0.75 

0 

0.875 

0 

B 

0.75 

0 

0.813 

0.75 

0.875 

0 

0.875 

0.875 

C 

0 

0 

0.25 

0 

0.625 

0.125 

0.688 

0.675 

D 

0.875 

1 

1 

0.875 

1 

1 

1 

1 

E 

0 

0 

0 

0 

1 

1 

1 

1 

F 

0 

0 

0 

0 

0 

0.313 

0.875 

0 

G 

0 

0 

0 

0 

0 

0 

0.75 

0 

missions,  and  two  “technology  levels”,  which  affects  the 
performance  of  different  propulsion  types.  Three  changeability 
usage  strategies  will  be  considered:  maximize  utility ,  maximize 
efficiency ,  and  survive  (transition  only  if  design  will  fail 
otherwise).  These  are  simple  strategies  but  represent  a  broad 
range  of  potential  ways  to  utilize  changeability  and  will  likely 
reflect  different  valuable  changeability  between  the  designs  of 
interest. 


robustness  across  the  epoch  space.  Looking  at  this  metric,  it 
appears  that  designs  D  and  E  are  the  most  desirable  options, 
with  excellent  scores  across  all  strategies  in  efNPT.  Figure  2 
plots  this  data  in  bar  chart  form,  which  can  provide  a  useful 
visual  for  seeing  what  designs  perform  well  across  which 
strategies  and  also  what  strategies  lend  themselves  to  more 
efficient  administration  of  the  system  with  which  designs. 


First,  eNPT  and  efNPT  are  used  to  scan  the  designs  for 
their  changeability- enabled  robustness ,  the  ability  to  remain 
valuable  over  variable  contexts  considering  all  planned  design 
changes,  which  considers  both  passive  robustness  and 
changeability  simultaneously.  Table  III  displays  the  results  of 
these  metrics  for  the  designs  of  interest  with  all  of  the 
strategies.  A  few  results  are  immediately  apparent.  For  one, 
considering  changeability  does  not  always  increase  Pareto 
Trace;  as  mentioned  previously,  some  strategies,  such  as 
maximize  utility ,  will  frequently  sacrifice  cost  efficiency  in  the 
name  of  another  goal  (here,  increasing  utility).  On  the  other 
hand,  the  maximize  efficiency  strategy  does  always  score  at 
least  as  well  as  the  “do  nothing”  NPT,  because  it  will  never 
sacrifice  proximity  to  the  Pareto  front  during  a  change.  It  is 
also  apparent  that  the  level  1  and  2  DFC  designs  (E,F,G)  do  not 
improve  from  NPT  to  eNPT,  because  they  have  a  fixed  cost 
increase  associated  with  changeability  that  distances  them  from 
the  Pareto  front  regardless  of  where  they  transition  to. 
However,  when  allowing  for  a  5%  fuzzy  margin,  these  designs 
see  a  distinct  improvement  between  fNPT  and  efNPT, 
particularly  design  E  which  has  a  perfect  score  of  one  in  every 
strategy.  This  is  capturing  information  as  desired:  the  highly 
changeable  designs,  while  never  strictly  (0%)  efficient,  can 
leverage  their  changeability  to  significantly  improve  value 


Effective  Fuzzy  Normalized  Pareto  Trace  (efNPT) 


Figure  2.  Space  Tug  efNPT  Bar  Chart 

Using  eNPT  and  efNPT  has  provided  an  understanding  of 
the  total  performance,  robustness  and  changeability  influenced, 
of  the  designs.  These  statistics  are  followed  up  with  an 
investigation  of  FPS,  which  isolates  the  value  added  by 
changeability.  Remember,  FPS  is  calculated  for  each  design  in 
each  epoch,  where  the  magnitude  value  of  changeability  is 
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found  in  the  score  for  each  epoch,  and  the  counting  value  is 
found  in  aggregate  across  the  epochs. 

Like  eNPT,  FPS  is  calculated  for  each  strategy  separately. 
The  preferred  way  to  view  FPS  data  is  with  a  distribution  of  the 
epoch  scores,  accompanied  by  a  table  of  the  order  statistics  of 
the  distributions  for  each  design.  Order  statistics  are  preferred 
to  mean  and  standard  deviation,  because  the  distributions  are 
frequently  irregular  and  median  is  a  vastly  superior  indicator  of 
central  tendency  for  them.  Figure  3  shows  the  distributions  for 
the  maximize  utility  strategy:  the  distributions  for  the  other 
strategies  are  not  displayed  for  conciseness.  However,  the  FPS 
order  statistics  for  the  designs  of  interest  in  each  strategy  are 
included  in  Table  IV.  If  a  design  is  invalid  and  cannot 
transition  to  a  valid  design,  its  score  becomes  -101  for  that 
epoch:  an  index  that  is  worse  than  the  worst  possible  FPN. 
Similarly,  if  an  invalid  design  becomes  valid,  its  “initial  FPN” 
(which  is  actually  undefined,  as  it  is  not  in  the  tradespace)  for 
the  purposes  of  calculating  FPS  is  treated  as  101. 

The  FPS  data  confirms  a  number  of  fundamental 
conjectures  about  the  Space  Tug  system.  For  one,  it  is  again 
clear  that  the  maximize  efficiency  strategy  does  not  allow  for 
transitions  leading  to  a  worsening  of  FPN:  these  transitions 
would  score  negative  FPS,  and  the  only  negative  FPS  scores 
reported  for  that  strategy  are  -101  scores  for  invalid  designs1. 
The  maximize  utility  strategy  tends  to  create  small  negative 
FPS  transitions,  with  only  the  occasional  large  boost,  in  this 
case  to  designs  A  and  F.  The  switch  from  the  utility  to  the 
efficiency  strategy  swaps  most  of  those  negative  FPS 
transitions  for  no-change  decisions,  as  apparent  from  the  large 
number  of  zeros.  This  is  due  in  part  to  the  selected  designs 
being  naturally  passively  efficient,  as  that  was  one  of  the 
selection  criteria.  Similarly,  the  survive  strategy  has  even  more 
no-change  decisions,  with  the  exception  of  design  A,  which  is  a 
bare-bones  design  with  minimal  fuel  that  must  change  to 
remain  functional. 

As  an  example  of  how  to  interpret  an  FPS  distribution  plot, 
Figure  3  displays  that: 

1.  Design  A  derives  very  large  value  from  a  few 
changes  (farthest  right  tail) 

2.  Design  F  most  consistently  generates  positive  value 
via  changeability  (most  weight  to  the  right  of  zero) 

3.  Designs  D,  E,  and  G  are  the  most  passively  robust 
(largest  spikes  at  zero  indicating  no  transition) 

4.  Designs  A,  B,  and  C  sacrifice  small  amounts  of 
efficiency  for  more  utility  (significant  weight  slightly 
left  of  zero) 

Overall,  the  plot  can  provide  a  quick-scan  grasp  of  the 
difference  in  valuable  changeability  between  the  designs  of 


1  The  -48  score  for  design  G’s  first  quartile  is  a  mathematical  artifact 
created  by  averaging  the  fourth  and  fifth  epochs’  performances  between  -101 
and  5,  and  does  not  represent  an  actual  -48  FPS  transition.  The  same  applies 
for  the  -50.5  first  quartile  scores  for  design  G  in  the  other  strategies:  it  is 
invalid  in  exactly  one  quarter  of  the  epochs. 


TABLE  IV.  Space  Tug  FPS  Order  Statistics 


Maximize  Utility  FPS  Order  Statistics 

Design 

Min 

1st  Quart 

Median 

3rd  Quart 

Max 

A 

-101 

-19 

-13 

-8 

93 

B 

-101 

-25.5 

-13.5 

-6 

-2 

C 

-10 

-9 

-6.5 

-1 

2 

D 

0 

0 

0 

0 

1 

E 

-3 

0 

0 

0 

0 

F 

-4 

6 

9 

28 

43 

G 

-101 

-50.5 

0 

0 

0 

Maximize  Efficiency  FPS  Order  Statistics 

Design 

Min 

1st  Quart 

Median 

3rd  Quart 

Max 

A 

-101 

0 

0 

0 

101 

B 

-101 

0 

0 

0 

4 

C 

0 

0 

1 

3 

9 

D 

0 

0 

0 

0 

1 

E 

0 

0 

0 

0 

0 

F 

9 

13 

18 

41 

52 

G 

-101 

-48 

8 

14 

30 

Survive  FPS  Order  Statistics 

Design 

Min 

1st  Quart 

Median 

3rd  Quart 

Max 

A 

-101 

-21 

-16.5 

-12 

85 

B 

-101 

0 

0 

0 

0 

C 

0 

0 

0 

0 

0 

D 

0 

0 

0 

0 

0 

E 

0 

0 

0 

0 

0 

F 

0 

0 

0 

0 

0 

G 

-101 

-50.5 

0 

0 

0 
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Figure  4.  Space  Tug  Design  ARI  Plot 


interest  that  is  difficult  to  process  out  of  the  tables  alone.  In 
this  case,  similar  insights  can  be  had  from  the  other 
distributions.  There  are  other  potential  views  with  the  ability  to 
assist  in  the  information  absorption  process.  For  example, 
coloring  the  tables  like  heat  maps  can  provide  a  more  distinct 
indicator  of  where  positive  (green)  and  negative  (red) 
transitions  occur.  Also,  the  tables  can  alternatively  be 
presented  in  plot  form  as  a  stacked  box-and-whisker  plot, 
displaying  the  minimum,  median,  maximum  and  quartiles 
visually  for  each  design,  separated  on  the  on  vertical  axis. 

Figure  4  is  an  example  ARI  plot  for  the  entire  tradespace. 
Recall  that  ARI  is  used  to  evaluate  the  effectiveness  of  the 
different  change  mechanisms  in  general,  thus  it  is  better  to 
examine  all  designs  rather  than  just  a  subset  of  the  designs. 
Also,  since  the  designs’  rank  order  rearranges  with  the 
preference  changes  in  the  different  epochs,  ARI  changes  in 
each  epoch;  Figure  4  is  an  example  for  Epoch  1,  with  the 
vertical  axis  showing  ARI  normalized  by  the  size  of  the  design 
space.  For  this  case  study,  it  happens  that  this  plot  looks  very 
similar  across  all  epochs,  so  the  conclusions  drawn  from  this 
example  will  serve  in  general. 

There  is  one  main  conclusion  able  to  be  drawn  from  the 
ARI  plot.  There  is  a  different  dominant  change  mechanism 
(“Rule”)  for  each  of  the  three  DFC  levels  (which  split  the 
tradespace  evenly  in  thirds):  the  fuel  tank  swap  for  level  0 
(Rule  2),  the  reduced-cost  version  of  the  same  rule  for  level  1 
(Rule  4),  and  the  refuel  in  orbit  for  level  2  (Rule  6),  which  all 
generate  more  than  twice  as  high  a  rank  increase  as  any  other 
mechanisms.  This  suggests  that  the  largest  utility  gains  created 
by  any  mechanisms  in  the  space  are  those  that  increase  the  fuel 
level  of  low- fuel  designs.  This  is  why  the  plot  is  jagged,  as  the 
high-fuel  designs  are  unable  to  exploit  those  mechanisms  for 
gain. 

VI.  Summary  and  Extensions 

Changeability  is  a  desirable  trait  in  engineering  systems,  as 
it  can  allow  them  to  avoid  risk  and  seize  opportunity. 
However,  a  number  of  mathematical  challenges  (dataset 
independence,  universality,  capturing  magnitude  and  counting 
value  of  transitions)  have  hindered  the  development  of  metrics 


to  justify  investment  in  changeability,  especially  when 
compared  to  the  older  and  better-understood  design  approach 
of  passive  robustness.  Overcoming  these  challenges  is  likely  to 
require  a  set  of  metrics  designed  to  properly  quantify  the  value 
of  changeability  together.  This  paper  has  discussed  the  aptness 
of  Epoch-Era  Analysis  to  structure  this  problem,  and  posited  a 
new  set  of  metrics  (eNPT/efNPT,  FPN,  FPS,  and  ARI) 
designed  to  perform  this  task  using  a  stakeholder-specified 
strategy.  An  example  case  using  a  Space  Tug  demonstrated  the 
application  of  these  metrics  in  the  multi-epoch  domain  and  the 
types  of  insights  about  valuable  changeability  in  an  engineering 
system  can  be  gained  from  their  use. 

Among  the  most  definitive  results  of  multi-epoch  analysis 
in  this  case  study  were  the  identification  of  designs  D  and  E  as 
extremely  passively  robust  (with  D  slightly  dominating  E  in 
cost  and  utility)  and  design  F  as  extremely  valuably 
changeable.  However,  it  is  difficult  to  directly  compare  these 
designs,  as  the  means  in  which  they  are  identified  are  on 
different  scales,  and  because  some  of  the  changeability  value  of 
F  is  obscured  without  evaluating  lifetime  behavior.  Multi¬ 
epoch  analysis  is  limited  in  that  it  does  not  consider  time-order 
effects  of  the  system  executing  changes  sequentially,  nor  does 
it  calculate  full  system  lifecycle  properties  such  as  lifetime  cost 
or  accumulated  utility  or  expected  number  of  change 
mechanism  uses,  which  are  typically  much  harder  to  predict  for 
highly  changeable  designs.  These  performance  metrics  can  be 
extracted  by  using  the  era  domain  of  Epoch-Era  Analysis,  and 
thus  it  is  recommended  that  multi-epoch  and  era  analysis  be 
performed  together  in  order  to  generate  as  much  information  on 
the  system  as  possible.  Without  evaluating  these  factors,  from 
which  changeable  designs  derive  much  of  their  potential 
advantages  over  passive  designs,  it  is  difficult  to  accurately 
compare  the  value  of  passively  versus  actively  robust  designs; 
era  analysis  allows  this  comparison.  Despite  this  limitation, 
multi-epoch  analysis  has  two  advantages  over  era  analysis:  (1) 
reduced  implementation  time  in  manpower  and  computation, 
and  (2)  fewer  assumptions,  as  the  era  constructor  must  make 
assumptions  about  the  duration  and  ordering  probabilities  of 
each  epoch,  which  may  lead  to  unreliable  results  if  not  chosen 
carefully.  For  more  detail  on  era  analysis,  consult  this  piece’s 
companion  paper  [12]. 
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